Evidence-Based Regularization for Neural Networks

Numerous approaches address over-fitting in neural networks: by imposing a penalty on the parameters of network (L1, L2, etc.); changing stochastically (drop-out, Gaussian noise, or transforming input data (batch normalization, etc.). In contrast, we aim to ensure that minimum amount supporting evidence is present when fitting model training data. This, at single neuron level, equivalent ensuring both sides separating hyperplane (for standard artificial neuron) have number points, noting these points need not belong same class for inner layers. We firstly benchmark results this approach Fashion-MINST dataset, comparing it various regularization techniques. Interestingly, note nudging each divide, least part, its data, resulting networks make use neuron, avoiding completely one side (which constant into next layers). To illustrate point, study prevalence saturated nodes throughout training, showing neurons are activated more frequently and earlier using approach. A direct consequence improved activation deep now easier train. This crucially important topology known priori often remains stuck suboptimal local minima. demonstrate property increasing depth (and width); most will result increasingly frequent failures (over different random seeds), whilst proposed evidence-based significantly outperforms ability train networks.